[BUG] ExecuteMonitor inserting metadata doc during dry run by petardz · Pull Request #758 · opensearch-project/alerting

petardz · 2023-01-16T18:36:59Z

Issue #, if available:

Description of changes:

Fixed ExecuteMonitor Action inserting metadata doc during dryrun

CheckList:
[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

lezzago · 2023-01-17T19:29:48Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt

                                log.info("Central Percolation index ${ScheduledJob.DOC_LEVEL_QUERIES_INDEX} created")
                            }
-                            val (metadata, _) = MonitorMetadataService.getOrCreateMetadata(monitor)
+                            val (metadata, _) = MonitorMetadataService.getOrCreateMetadata(monitor, skipIndex = true)


This should only be true if the dryrun param is true

even if monitorId is null ?

I think if monitorId is null we definitely can't save metadata (there's no monitor to tie it to).

Dryrun can be true for existing monitor and we take that into account before indexing metadata inside DocumentLevelMonitorRunner here

Oh I missed that part of the logic, good call out.

lezzago · 2023-01-17T19:39:22Z

alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt

-                    if (e.message?.contains("Limit of total fields") == true) {
+                    val unwrappedException = ExceptionsHelper.unwrapCause(e) as Exception
+                    log.debug("exception after rollover queryIndex index: $targetQueryIndex exception: ${unwrappedException.message}")
+                    if (unwrappedException.message?.contains("Limit of total fields") == true) {


Will this be changed to false from true ever?

codecov-commenter · 2023-01-17T23:25:06Z

Codecov Report

Merging #758 (95d3be7) into main (8084943) will increase coverage by 0.27%.
The diff coverage is 83.33%.

@@             Coverage Diff              @@
##               main     #758      +/-   ##
============================================
+ Coverage     75.27%   75.55%   +0.27%     
  Complexity      116      116              
============================================
  Files           125      125              
  Lines          6828     6835       +7     
  Branches       1024     1027       +3     
============================================
+ Hits           5140     5164      +24     
+ Misses         1159     1134      -25     
- Partials        529      537       +8

Impacted Files	Coverage Δ
.../org/opensearch/alerting/MonitorMetadataService.kt	`65.38% <66.66%> (+0.30%)`	⬆️
.../opensearch/alerting/DocumentLevelMonitorRunner.kt	`77.81% <100.00%> (+0.23%)`	⬆️
...lerting/transport/TransportExecuteMonitorAction.kt	`82.08% <100.00%> (ø)`
...rch/alerting/opensearchapi/OpenSearchExtensions.kt	`50.00% <0.00%> (-1.62%)`	⬇️
...lin/org/opensearch/alerting/alerts/AlertIndices.kt	`74.27% <0.00%> (+0.41%)`	⬆️
...destinationmigration/DestinationConversionUtils.kt	`70.00% <0.00%> (+1.11%)`	⬆️
...in/org/opensearch/alerting/MonitorRunnerService.kt	`81.63% <0.00%> (+2.04%)`	⬆️
.../kotlin/org/opensearch/alerting/util/IndexUtils.kt	`72.91% <0.00%> (+2.08%)`	⬆️
.../alerting/transport/TransportIndexMonitorAction.kt	`57.89% <0.00%> (+2.96%)`	⬆️
...ing/model/destination/DestinationContextFactory.kt	`78.57% <0.00%> (+3.57%)`	⬆️
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt

* execute monitor bugfix Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * added IT Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * fixed created retval when skipIndex=true Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> --------- Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> (cherry picked from commit ce7094a)

* execute monitor bugfix Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * added IT Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * fixed created retval when skipIndex=true Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> --------- Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> (cherry picked from commit ce7094a) Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com>

…h-project#758) * execute monitor bugfix Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * added IT Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * fixed created retval when skipIndex=true Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> --------- Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* Added exception check once the .opendistro-alerting-config index is b… (#650) * Added exception check once the .opendistro-alerting-config index is being created During .opendistro-alerting-config index creation, if ResourceAlreadyExists exception is being raised, the flow will check first if the index is in yellow state and then it will re-try to index monitor Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Formating of the file fixed Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * refactored DeleteMonitor Action to be synchronious (#628) (#630) * refactored DeleteMonitor Action to be synchronious Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> * [Backport 2.x] QueryIndex rollover when field mapping limit is reached (#729) Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * Mappings fix backport 2.x (#730) * Added support for "nested" mappings (#645) * example Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * fixed updating mappings for queryIndex Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * mappings traversal bug fix (#669) Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * Added unwrapping exception from core; added more debug logs (#728) Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * Add DataSources test for future backports Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * fix percolator mapping error when having field name 'type' (#726) Signed-off-by: Raj Chakravarthi <raj@icedome.ca> * [BUG] ExecuteMonitor inserting metadata doc during dry run (#758) * execute monitor bugfix Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * added IT Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * fixed created retval when skipIndex=true Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> --------- Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * fix for ERROR alert state generation in doc-level monitors (#768) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Adjusting max field index setting dynamically for query index (#776) * added adjusting max field index setting dynamicly for query index Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * Multiple indices support in DocLevelMonitorInput (#784) Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * Doc transform 2.x backport (#853) * conflict resovle - backport from main to 2.x Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * fixed module class names Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> --------- Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * Update config index schema if needed at the start of each monitor execution (#849) * Update config index schema if needed at the start of each monitor execution Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> * Mappings parsing fix (#851) * fixed mappings parsing when field name named "properties" exists in mappings Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * message typo fix Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> --------- Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * [Backport 2.x] Notification security fix (#861) * Notification security fix (#852) * added injecting whole user object in threadContext before calling notification APIs so that backend roles are available to notification plugin Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * compile fix Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * refactored user_info injection to use InjectSecurity Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> * ktlint fix Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> --------- Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> (cherry picked from commit e0b7a5a) * remove unneeded import Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> --------- Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> * Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names. (#894) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * fixed security tests (#484) (#794) * fixed security tests Signed-off-by: Raj Chakravarthi <raj@icedome.ca> (cherry picked from commit c51940f) --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Raj Chakravarthi <raj@icedome.ca> Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: Stevan Buzejic <30922513+stevanbz@users.noreply.github.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: RAJ CHAKRAVARTHI <49325334+raj-chak@users.noreply.github.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: RAJ CHAKRAVARTHI <raj@icedome.ca>

petardz requested a review from a team January 16, 2023 18:37

petardz changed the title ~~Monitor execute bugfix~~ ExecuteMonitor inserting metadata doc during dry run Jan 17, 2023

petardz changed the title ~~ExecuteMonitor inserting metadata doc during dry run~~ [BUG] ExecuteMonitor inserting metadata doc during dry run Jan 17, 2023

lezzago requested changes Jan 17, 2023

View reviewed changes

petardz added 3 commits January 18, 2023 00:51

execute monitor bugfix

9e92fb5

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

added IT

e61b19c

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

fixed created retval when skipIndex=true

95d3be7

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

petardz force-pushed the alerting-fix branch from 1122500 to 95d3be7 Compare January 17, 2023 23:51

lezzago reviewed Feb 3, 2023

View reviewed changes

alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt Show resolved Hide resolved

lezzago added backport 2.x backport 2.5 labels Feb 3, 2023

lezzago approved these changes Feb 3, 2023

View reviewed changes

AWSHurneyt approved these changes Feb 3, 2023

View reviewed changes

AWSHurneyt merged commit ce7094a into opensearch-project:main Feb 3, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 3, 2023

[Backport 2.x] [BUG] ExecuteMonitor inserting metadata doc during dry run #777

Merged

opensearch-trigger-bot bot mentioned this pull request Feb 3, 2023

[Backport 2.5] [BUG] ExecuteMonitor inserting metadata doc during dry run #778

Merged

AWSHurneyt mentioned this pull request Jun 6, 2023

[Backport 2.5] Manual backports of several PRs #957

Merged

lezzago mentioned this pull request Jun 8, 2023

[Backport 2.3] Manual backports of several PRs #960

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] ExecuteMonitor inserting metadata doc during dry run#758

[BUG] ExecuteMonitor inserting metadata doc during dry run#758
AWSHurneyt merged 3 commits intoopensearch-project:mainfrom
petardz:alerting-fix

petardz commented Jan 16, 2023 •

edited

Loading

Uh oh!

lezzago Jan 17, 2023

Uh oh!

petardz Jan 17, 2023

Uh oh!

petardz Jan 17, 2023 •

edited

Loading

Uh oh!

lezzago Feb 3, 2023

Uh oh!

lezzago Jan 17, 2023

Uh oh!

codecov-commenter commented Jan 17, 2023 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

petardz commented Jan 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lezzago Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

petardz Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

petardz Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lezzago Feb 3, 2023

Choose a reason for hiding this comment

Uh oh!

lezzago Jan 17, 2023

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

petardz commented Jan 16, 2023 •

edited

Loading

petardz Jan 17, 2023 •

edited

Loading

codecov-commenter commented Jan 17, 2023 •

edited

Loading